Information Processing Device, Operation Input Method And Operation Input Program

ABSTRACT

Disclosed herein is an information processing device including a contact detection unit configured to detect a contact operation for an operation surface, an approach detection unit configured to detect an approach operation for the operation surface, and a control unit configured to recognize the contact operation and the approach operation as a series of operation inputs when the contact operation and the approach operation are continuously detected.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 15/806,631, filed Nov. 8, 2017 which is a continuation of U.S.application Ser. No. 15/332,213, filed Oct. 24, 2016, issued as U.S.Pat. No. 9,841,838 on Dec. 12, 2017 which is a continuation of U.S.application Ser. No. 14/033,997, filed Sep. 23, 2013, issued as U.S.Pat. No. 9,507,469 on Nov. 29, 2016 which is a continuation of U.S.application Ser. No. 12/930,590, filed Jan. 11, 2011, issued as U.S.Pat. No. 8,581,864 on Nov. 12, 2013 which claims priority from JapanesePatent Application No. JP 2010-009183 filed in the Japanese PatentOffice on Jan. 19, 2010, the entire contents of which are incorporatedherein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an information processing device, anoperation input method and an operation input program and is suitablyapplicable to, for example, an information processing device having atouch panel.

2. Description of the Related Art

In recent years, an information processing device having a touch panelhas come into wide use. In an information processing device having atouch panel, contact of an indicator (finger, touch pen, or the like)with the touch panel is detected and received as an operation input soas to execute various processes.

Recently, an information processing device capable of detecting approachin addition to contact of an indicator with a touch panel has beenproposed (for example, Japanese Unexamined Patent ApplicationPublication No. 2006-302126). In this information processing device,even when a finger which is in contact with the touch panel is separatedfrom the touch panel, if the finger is close to the touch panel, aprocess performed when the finger is in contact with the touch panel iscontinuously performed.

For example, when a button displayed on a screen is long pressed,although the finger is separated from the touch panel partway byvibration or the like, the same process when the button is long pressed,that is, the process intended by a user may be executed.

SUMMARY OF THE INVENTION

However, in the information processing device having the touch panel ofthe related art, fundamentally, an operation input is performed bybringing the indicator into contact with the touch panel.

If approach is selected as an object for the operation input in additionto the contact of the indicator with the touch panel, it is possible toperform various operation inputs compared with the related art.

It is desirable to provide an information processing device, anoperation input method and an operation input program, which is capableof performing various operation inputs, compared with the related art.

According to an embodiment of the present invention, there is providedan information processing device including: a contact detection unitconfigured to detect a contact operation for an operation surface; anapproach detection unit configured to detect an approach operation forthe operation surface; and a control unit configured to recognize thecontact operation and the approach operation as a series of operationinputs when the contact operation and the approach operation arecontinuously detected.

By recognizing the continuous contact operation and approach operationas the series of operation inputs, it is possible to increase the kindsof the operation inputs compared with the case of selecting only thecontact operation as an object for an operation input.

According to the present invention, by recognizing the continuouscontact operation and approach operation as the series of operationinputs, it is possible to increase the kinds of the operation inputscompared with the case of selecting only the contact operation as anobject for an operation input. Therefore, it is possible to realize aninformation processing device, an operation input method and anoperation input program, which is capable of performing variousoperation inputs compared with the related art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the functional configuration of aninformation processing device according to the present embodiment.

FIG. 2 is a schematic diagrammatic view showing the appearanceconfiguration of a mobile terminal of a detailed example of the presentinvention.

FIG. 3 is a block diagram showing the hardware configuration of a mobileterminal.

FIG. 4 is a schematic diagrammatic view showing a change of an outputvalue of an electrostatic sensor.

FIG. 5 is a schematic diagrammatic view serving to describe adetermination of a contact portion, an approach portion and anon-approach portion.

FIG. 6 is a schematic diagrammatic view serving to describe the displayof a web browser screen.

FIG. 7 is a schematic diagrammatic view showing an example of a touchoperation and an approach operation.

FIG. 8 is a schematic diagrammatic view serving to describe an example(1) of performing an approach operation within a predetermined time Tafter a touch operation.

FIG. 9 is a schematic diagrammatic view serving to describe an exampleof performing an approach operation when exceeding a predetermined timeT after a touch operation.

FIG. 10 is a schematic diagrammatic view serving to describe an example(2) of performing an approach operation within a predetermined time Tafter a touch operation.

FIG. 11 is a flowchart illustrating a procedure of an operation inputreception process.

FIG. 12 is a schematic diagrammatic view showing an operation inputexample (1) according to another embodiment.

FIG. 13 is a schematic diagrammatic view showing an operation inputexample (2) according to another embodiment.

FIG. 14 is a schematic diagrammatic view showing an operation inputexample (3) according to another embodiment.

FIG. 15 is a schematic diagrammatic view showing an operation inputexample (4) according to another embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, a best mode (hereinafter, referred to as an embodiment) ofthe invention will be described. The description will be given in thefollowing order.

1. Embodiment

2. Other Embodiments

1. Embodiment 1-1. Overview of Embodiment

First, the overview of an embodiment will be described. After theoverview of the embodiment is described, detailed examples of thepresent invention will be given.

In FIG. 1, a reference numeral 1 denotes an information processingdevice. In this information processing device 1, a contact detectionunit 2 for detecting a contact operation with respect to an operationsurface (for example, an operation surface of a touch panel) and anapproach detection unit 3 for detecting an approach operation withrespect to the operation surface are provided. In the informationprocessing device 1, a control unit 4 for recognizing the contactoperation and the approach operation as a series of operation inputswhen the contact operation and the approach operation are continuouslydetected is provided.

The information processing device 1 recognizes the continuous contactoperation and approach operation as the series of operation inputs so asto increase the kinds of the operation inputs compared with the case ofselecting only the contact operation as an object for an operationinput.

In more detail, the control unit 4 recognizes the contact operation andthe approach operation as the series of operation inputs when thecontact operation and the approach operation are continuously detectedwithin a predetermined time. When the contact operation and the approachoperation are continuously detected after exceeding the predeterminedtime, the contact operation and the approach operation are recognized asindividual operation inputs.

At this time, the control unit 4 controls the execution of apredetermined process based on a predetermined contact operationdetected by the contact detection unit 2 and controls a restorationprocess to a state before the execution of the predetermined processwhen the approach detection unit detects a predetermined approachoperation within a predetermined time from the detection of thepredetermined contact operation.

The control unit 4 may control the execution of a predetermined processbased on a predetermined contact operation detected by the contactdetection unit 2 and continue to execute the predetermined process whenthe approach detection unit 3 detects another approach operation withina predetermined time from the detection of the predetermined contactoperation. At this time, another approach operation is, for example, anoperation for separating an indicator from an operation surface andstopping the indicator after the contact operation.

The control unit 4 may control the execution of a predetermined processbased on predetermined motion of a contact operation detected by thecontact detection unit 2 and execute the predetermined process using aparameter different from a parameter used upon the execution based onthe contact operation when the contact detection unit 3 detects thepredetermined motion of the approach operation within a predeterminedtime from the detection of the contact operation.

The control unit 4 may execute control based on a certain direction anda contact operation, when the contact detection unit 2 detects a contactoperation within a predetermined time after the approach detection unit3 detects an approach operation for moving an indicator in a certaindirection.

The control unit 4 may recognize a contact operation and an approachoperation as a series of operation inputs, when the approach detectionunit 3 detects an approach operation within a predetermined range from acontact position within a predetermined time after the contact detectionunit 2 detects a contact operation.

Detailed examples of the information processing device 1 having theabove configuration will now be described in detail.

1-2. Detailed Example of Embodiment 1-2-1. Appearance Configuration ofMobile Terminal

Next, the appearance configuration of a mobile terminal 100 which is adetailed example of the above-described information processing device 1will be described using FIG. 2.

The mobile terminal 100 is formed to be grasped by one hand, by openablyand closably connecting substantially flat rectangular first and secondcasings 101 and 102 through hinge portions 103A and 103B.

A first rectangular touch screen 104 is provided in a central portion ofa front surface 101A of the first casing 101. In addition, a secondtouch screen 105 having the same shape and size as the first touchscreen 104 is provided in a central portion of a front surface 102A ofthe second casing 102.

Each of the first touch screen 104 and the second touch screen 105includes a liquid crystal panel and a thin transparent touch panelcovering a display surface of the liquid crystal panel. This touch panelis of a capacitance type and corresponds to a multitouch panel.

The mobile terminal 100 recognizes a contact operation and an approachoperation of a finger (a touch pen or the like) on each of the firsttouch screen 104 and the second touch screen 105 as operation inputs. Inaddition, the contact operation is an operation (also referred to as atouch operation) performed by touching a touch screen with a finger andthe approach operation is an operation performed by enabling a finger toapproach a touch screen without touching the touch screen.

The mobile terminal 100 is used in a state in which the first casing 101is located at an upper side and the second casing 102 is located at alower side. In general, a variety of information is displayed in a statein which the first touch screen 104 is an upper screen and the secondtouch screen 105 is a lower screen.

A variety of operation keys 106 such as a power button is provided onthe front surface 102A of the second casing 102 of the mobile terminal100 at both sides of the second touch screen 105.

1-2-2. Hardware Configuration of Mobile Terminal

Next, the hardware configuration of the mobile terminal 100 will bedescribed using FIG. 3. In the mobile terminal 100, a CPU 110 develops aprogram stored in a non-volatile memory 111 in a RAM 112, reads theprogram, executes a variety of processes according to the program, andcontrols each unit. In addition, the CPU is an abbreviation for aCentral Processing Unit and the RAM is an abbreviation for a RandomAccess Memory.

When a touch operation or an approach operation for the first touchscreen 104 and the second touch screen 105 is recognized, the CPU 110receives the operation as an operation input and executes a processaccording to the operation input.

The first touch screen 104 includes a first liquid crystal panel 104Aand a first touch panel 104B and the second touch screen 105 includes asecond liquid crystal panel 105A and a second touch panel 105B.

The first liquid crystal panel 104A and the second liquid crystal panel105A display a variety of information as the upper screen and the lowerscreen, respectively.

Each of the first touch panel 104B and the second touch panel 105B is acapacitance touch panel, as described above.

The first touch panel 104B has a plurality of electrostatic sensors (notshown) arranged in a lattice shape on an operation surface. An outputvalue of each of the plurality of electrostatic sensors is changedaccording to capacitance changed as a conductor such as a finger comesclose to the operation surface.

In practice, as shown in FIG. 4, it is assumed that a finger approachesthe operation surface of the first touch panel 104B. At this time, theoutput value of the electrostatic sensor located just under the fingeris, for example, changed to “10” when a distance between the operationsurface and the finger is 30 mm, “20” when the distance is 15 mm, “40”when the distance is 5 mm, and “80 as a maximum when the finger makescontact with the operation surface.

The CPU 110 acquires the changed output value of each electrostaticsensor and the position of each electrostatic sensor from the firsttouch panel 104B. The CPU 110 discriminates among a portion with whichthe finger contacts, a portion to which the finger approaches, and aportion with which the finger does not contact or to which the fingerdoes not approach, on the operation surface of the first touch panel104B based on the output value of each electrostatic sensor and theposition of each electrostatic sensor. In addition, the portion withwhich the finger contacts is referred to as a contact portion, theportion to which the finger approaches is referred to as an approachportion, and the portion with which the finger does not contact or towhich the finger does not approach is referred to as a non-approachportion.

In detail, as shown in FIG. 5, the CPU 110 discriminates a portion, inwhich the output value of the electrostatic sensor is equal to orgreater than “50”, on the operation surface of the first touch panel104B as the contact portion. In addition, the CPU 110 discriminates aportion in which the output value is equal to or greater than “20” andis less than “50” as the approach portion and discriminates a portion inwhich the output value is less than “20” as the non-approach portion.

The CPU 110 discriminates the contact portion, the approach portion andthe non-approach portion on the operation surface of the first touchpanel 104B so as to specify whether the finger contacts the operationsurface or approaches the operation surface. In addition, it is possibleto specify where the finger contacts or approaches the operationsurface. At this time, if a contact portion is present, the CPU 110recognizes, for example, a centroid or a center (a centroid or a centerof a ball of a finger which contacts the operation surface) of thecontact portion as a touch position. If only an approach portion ispresent, the CPU 110 recognizes, for example, a centroid or a center (acentroid or a center of a ball of a finger which approaches theoperation surface) as an approach position.

In addition, the CPU 110 acquires the output value of each electrostaticsensor from the first touch panel 104B every fixed time period anddiscriminates among the contact portion, the approach portion and thenon-approach portion so as to detect the transition of the touchposition and the approach position based on the contact portion, theapproach portion and the non-approach portion. The CPU 110 specifies themotion of the finger on the first touch panel 104B based on thetransition.

The CPU 110 recognizes and receives the touch operation and the approachoperation for the first touch screen 104 as the operation inputs basedon the specified motion of the finger on the first touch panel 104B.

The second touch panel 105B has a plurality of electrostatic sensors(not shown) arranged in a lattice shape on an operation surface, similarto the first touch panel 104B.

The CPU 110 acquires the output value of each electrostatic sensor fromthe second touch panel 105B every fixed time, discriminates among acontact portion, an approach portion and a non-approach portion on theoperation surface of the second touch panel 105B, and detects thetransition of a touch position and an approach position.

The CPU 110 specifies the motion of the finger on the second touch panel105B based on the transition and recognizes and receives the touchoperation and the approach operation for the second touch screen 105 asthe operation inputs based on the motion of the finger.

The CPU 110 recognizes and receives the touch operation and the approachoperation for the first touch screen 104 and the second touch screen 105as the operation inputs, and executes processes according to theoperation inputs.

The CPU 110 recognizes and receives a pressing operation of an operationkey 106 as an operation input and executes a process according to thisoperation input.

For example, it is assumed that, in a state in which a playback buttonfor playing back music data is displayed on the second touch screen 105,a user taps the playback button.

If such a touch operation is detected, the CPU 110 recognizes andreceives the touch operation as an operation input for playing back themusic data, reads the music data from the non-volatile memory 111, andsends the music data to a playback unit 113.

The playback unit 113 performs a playback process such as a decodingprocess, a digital-to-analog conversion process and an amplificationprocess with respect to the music data under the control of the CPU 110,obtains a sound signal, and outputs the sound signal to a headphoneterminal (not shown).

Thus, the user may listen to music using headphones connected to theheadphone terminal.

At this time, the CPU 110 acquires information such as a song title oran artist name from the read music data and displays the information,for example, on the first liquid crystal panel 104A of the first touchscreen 104A. Thus, it is possible to present the information about themusic which is being played back to the user.

For example, it is assumed that, in a state in which a browser icon forstarting up a web browser is displayed on the second touch screen 104, auser taps the browser icon.

If such a touch operation is detected, the CPU 110 recognizes andreceives the touch operation as an operation input for starting up theweb browser and starts up the web browser.

The CPU 110 displays a web browser screen on the first liquid crystalpanel 104A and the second liquid crystal panel 105A and receives pagedata of a web page from a server on a network through a networkinterface 114. The CPU 110 displays a page image based on the page dataon the web browser screen.

Thus, the user may browse the web page through the web browser screen.

The detailed hardware examples of the contact detection unit 2 and theapproach detection unit 3 of the information processing device 1described as the overview of the present embodiment are the first touchpanel 104B and the second touch panel 105B of the above-described mobileterminal 100. The detailed hardware example of the control unit 4 of theinformation processing device 1 is the CPU 110 of the mobile terminal100.

The mobile terminal 100 not only recognizes and receives the touchoperation and the approach operation as individual operation inputs, butalso recognizes and receives such operations as a series of operationinputs when such operations are continuously performed within apredetermined time.

Hereinafter, the operation inputs when the touch operation and theapproach operation are continuously performed will be described indetail.

1-3. Operation Input when Touch Operation and Approach Operation areContinuously Performed

As shown in FIG. 6, it is assumed that an upper half of the web browserscreen is displayed on the first touch screen 104 and a lower half ofthe web browser screen is displayed on the second touch screen 105.

At this time, as the contact operations and the approach operationsthere are pinch-in, pinch-out, shuffle and the like which are recognizedand received by the CPU 110 as the operation inputs.

Pinch-in refers to an operation for narrowing a space between twofingers (for example, a thumb and a forefinger). As shown in (A) of FIG.7, pinch-in includes contact pinch-in as a touch operation performed byenabling two fingers to touch a touch panel and approach pinch-in as anapproach operation performed by enabling two fingers to approach a touchpanel.

Pinch-out refers to an operation for widening a space between twofingers. As shown in (B) of FIG. 7, pinch-out includes contact pinch-outas a touch operation performed by enabling two fingers to touch a touchpanel and approach pinch-out as an approach operation performed byenabling two fingers to approach a touch panel.

Shuffle refers to an operation for drawing a zigzag with one finger. Asshown in (C) of FIG. 7, shuffle includes only approach shuffle as anapproach operation performed by enabling one finger to approach a touchpanel. In addition, in order to maintain compatibility with a generaloperation of the related art which performs only a touch operation as anoperation input, when one finger touches a touch panel so as to performshuffle, the CPU 110 does not recognize contact shuffle, but recognizesgeneral drag.

As an example of continuously performing the touch operation and theapproach operation, for example, approach shuffle may be performed aftercontact pinch-in.

In practice, as shown in FIG. 8, it is assumed that, at a certain timet1, contact pinch-in is performed as a touch operation with respect tothe first touch screen 104.

When the contact pinch-in is detected, the CPU 110 recognizes andreceives the contact pinch-in as an operation input for reducing a pageimage of a web browser screen, reduces the page image, and displays thereduced page image on the web browser screen.

At this time, the CPU 110 sets a reduction ratio of the page image, forexample, in a range of ½ to ⅓ of the movement amount of two fingers whenthe contact pinch-in is performed. The CPU 110 sets a value (½) obtainedby multiplying a current display scaling factor (for example, one time)of the page image by the set reduction ratio (for example, ½) as a newdisplay scaling factor (½) and reduces the page image.

Thereafter, it is assumed that, at a certain time t2, the user separatesthe fingers from the first touch screen 104 so as to perform approachshuffle as an approach operation.

When the approach shuffle is detected, the CPU 110 determines whether ornot the current time t2 is within a predetermined time T (about 1second) from the time t1 when the previous touch operation (contactpinch-in) is performed.

If the current time t2 when the approach shuffle is performed is withinthe predetermined time T from the time t1 when the previous touchoperation is performed, the CPU 110 determines that the current approachoperation (approach shuffle) is an operation associated with theprevious touch operation.

At this time, the CPU 110 recognizes and receives the approach shuffleas an operation input for returning the display scaling factor of thepage image to a value before reduction by the previous contact pinch-inand returns the display scaling factor of the page image to the value(that is, one time) before reduction.

In contrast, as shown in FIG. 9, it is assumed that the current time t2which the approach shuffle is performed exceeds the predetermined time Tfrom the time t1 when the previous touch operation is performed. In thiscase, the CPU 110 determines that the current approach operation(approach shuffle) is independent of the previous touch operation.

At this time, the CPU 110 recognizes and receives the approach shuffleas an operation input returned to the previous page image and displaysthe previous page image on the web browser screen instead of thecurrently displayed page image.

When the approach shuffle is performed within the predetermined time Tafter the contact pinch-in, the CPU 110 recognizes and receives theapproach shuffle as a series of operation inputs continued to theprevious contact pinch-in.

When the approach shuffle is performed after the predetermined time Tafter the contact pinch-in, the approach shuffle is recognized andreceived as an operation input independent of the previous contactpinch-in.

As another example of continuously performing the touch operation andthe approach operation, for example, approach shuffle may be performedafter contact pinch-out.

In practice, it is assumed that, at a certain time t1, contact pinch-outis performed as a touch operation with respect to the first touch screen104 (not shown) by the user.

When the contact pinch-out is detected, the CPU 110 recognizes andreceives the contact pinch-out as an operation input for enlarging apage image of a web browser screen, enlarges the page image, anddisplays the enlarged page image on the web browser screen.

At this time, the CPU 110 sets an enlargement ratio, for example, in arange of two times to three times the movement amount of two fingerswhen the contact pinch-out is performed. The CPU 110 sets a value (twotimes) obtained by multiplying a current display scaling factor (forexample, one time) of the page image by the set enlargement ratio (forexample, two times) as a new display scaling factor (two times) andenlarges the page image.

Thereafter, it is assumed that, at a certain time t2, the user separatesthe fingers from the first touch screen 104 so as to perform approachshuffle as an approach operation.

When the approach shuffle is detected, the CPU 110 determines whether ornot the current time t2 is within a predetermined time T (about 1second) from the time t1 when the previous touch operation (contactpinch-out) is performed.

If the current time t2 when the approach shuffle is performed is withinthe predetermined time T from the time t1 when the previous touchoperation is performed, the CPU 110 determines that the current approachoperation (approach shuffle) is an operation associated with theprevious touch operation.

At this time, the CPU 110 recognizes and receives the approach shuffleas an operation input for returning the display scaling factor of thepage image to a value before enlargement by the previous contactpinch-out and returns the display scaling factor of the page image tothe value (that is, one time) before enlargement.

In contrast, if the current time t2 which the approach shuffle isperformed exceeds the predetermined time T from the time t1 when theprevious touch operation is performed, the CPU 110 determines that thecurrent approach operation (approach shuffle) is independent of theprevious touch operation.

At this time, the CPU 110 recognizes and receives the approach shuffleas an operation input returned to the previous page image and displaysthe previous page image on the web browser screen instead of thecurrently displayed page image.

When the approach shuffle is performed within the predetermined time Tafter the contact pinch-out, the CPU 110 recognizes and receives theapproach shuffle as a series of operation inputs continued to theprevious contact pinch-out.

When the approach shuffle is performed after the predetermined time Tafter the contact pinch-out, the approach shuffle is recognized andreceived as an operation input independent of the previous contactpinch-out.

As another example of continuously performing the touch operation andthe approach operation, for example, approach pinch-out may be performedafter contact pinch-out.

In practice, as shown in FIG. 10, it is assumed that, at a certain timet1, contact pinch-out is performed as a touch operation with respect tothe first touch screen 104 by the user.

When the contact pinch-out is detected, the CPU 110 recognizes andreceives the contact pinch-out as an operation input for enlarging apage image of a web browser screen, enlarges the page image, anddisplays the enlarged page image on the web browser screen, as describedabove.

At this time, the CPU 110 sets an enlargement ratio, for example, in arange of two times to three times the movement amount of two fingerswhen the contact pinch-out is performed. The CPU 110 sets a value (twotimes) obtained by multiplying a current display scaling factor (forexample, one time) of the page image by the set enlargement ratio (forexample, two times) as a new display scaling factor (two times) andenlarges the page image.

Thereafter, it is assumed that, at a certain time t2, the user separatesthe fingers from the first touch screen 104 so as to perform approachpinch-out as an approach operation.

When the approach pinch-out is detected, the CPU 110 determines whetheror not the current time t2 is within a predetermined time T from thetime t1 when the previous touch operation (contact pinch-out) isperformed.

If the current time t2 when the approach pinch-out is performed iswithin the predetermined time T from the time t1 when the previous touchoperation is performed, the CPU 110 determines that the current approachoperation (approach pinch-out) is an operation associated with theprevious touch operation.

At this time, the CPU 110 recognizes and receives the approach pinch-outas an operation input for more finely setting the enlargement ratio ofthe page image enlarged by the previous contact pinch-out and morefinely sets the enlargement ratio of the page image.

At this time, the CPU 110 more finely sets the enlargement ratio, forexample, in a range of +0.2 times to +0.3 times of the movement amountof two fingers when the contact pinch-out is performed. The CPU 110 setsa value (2.3 times) obtained by adding the set enlargement ratio (forexample, +0.3 times) to a current display scaling factor (for example,two times) of the page image as a new display scaling factor (2.3 times)and enlarges the page image.

In contrast, if the current time t2 which the approach pinch-out isperformed exceeds the predetermined time T from the time t1 when theprevious touch operation is performed, the CPU 110 determines that thecurrent approach operation (approach pinch-out) is independent of theprevious touch operation. At this time, the CPU 110 does not receive theapproach pinch-out as an operation input.

When the approach pinch-out is performed within the predetermined time Tafter the contact pinch-out, the CPU 110 recognizes and receives theapproach pinch-out as an operation input associated with the previouscontact pinch-out.

As described up to now, the mobile terminal 100 recognizes and receivesnot only the touch operation but also the approach operation as theoperation input with respect to the first touch panel 104B and thesecond touch panel 105B.

Accordingly, in the mobile terminal 100, it is possible to performvarious operation inputs, compared with the case where only the touchoperation is recognized and received as the operation input.

In the mobile terminal 100, the approach operation is recognized andreceived as the operation input in addition to the touch operation. Inaddition, when the touch operation and the approach operation arecontinuously performed within the predetermined time T, these operationsare recognized and received as a series of operation inputs.

Therefore, in the mobile terminal 100, it is possible to recognize andreceive the touch operation and the approach operation as individualoperation inputs and to perform an operation input obtained by combiningsuch operations. As a result, it is possible to perform variousoperation inputs.

As described above, in the mobile terminal 100, when a specific approachoperation (for example, approach shuffle) is performed within thepredetermined time T after the touch operation, a process of returningto a state before the process of the touch operation is executed, isexecuted.

For example, even when a button for returning to an original state,which is displayed on the screen, is not touched, it is possible toreturn a state just before the touch operation by one approachoperation.

In the mobile terminal 100, after a touch operation forenlarging/reducing an image or the like, when the approach operation ofthe same motion as the touch operation is performed within thepredetermined time T, the same process (enlargement/reduction) isexecuted by a change ratio different from that of the touch operation.

Therefore, for example, after the image is roughly enlarged/reduced bythe touch operation, the size of the image may be finely adjusted by theapproach operation.

1-4. Procedure of Receiving Operation Input

Next, a detailed procedure (this is also referred to as an operationinput reception process) of recognizing and receiving an operation inputby the mobile terminal 100 will be described using the flowchart shownin FIG. 11. The operation input reception procedure RT1 shown in FIG. 11is a procedure executed according to a program stored in thenon-volatile memory 111 by the CPU 110 of the mobile terminal 100.

The CPU 110 starts the operation input reception procedure RT1, forexample, when the mobile terminal 100 is powered on, and transitions tostep SP1. In step SP1, the CPU 110 determines whether or not the contactof the finger with the first touch panel 104B or the second touch panel105B is detected.

For example, if a positive result is obtained in step SP1 by thedetection of the contact of the finger with the first touch panel 104B,the CPU 110 transitions to step SP2.

In step SP2, the CPU 110 detects a touch operation and transitions tonext step SP3. In step SP3, the CPU 110 determines whether or not thetouch operation (for example, the above-described contact pinch-in,contact pinch-out, or the like) corresponding to the operation input isdetected in step SP2 of the previous stage.

If the touch operation corresponding to the operation input is detectedin step SP2 of the previous stage and thus a positive result is obtainedin step SP3, the CPU 110 sets a timer for measuring the above-describedpredetermined time T and transitions to step SP4.

In step SP4, the CPU 110 recognizes and receives the touch operationdetected in step SP2 as an operation input and issues a command (forexample, a command for reducing a page image) corresponding to theoperation input. As a result, a process (for example, a process ofreducing the page image) corresponding to the command is executed. TheCPU 110 returns to step SP1 again. If a negative result is obtained inthe above-described step SP3 (that is, if the touch operation detectedin step SP2 is not the touch operation corresponding to the operationinput), the CPU 110 returns to step SP1 again.

If the contact of the finger is not detected and thus a negative resultis obtained in the above-described step SP1, the CPU 110 transitions tostep SP5.

In step SP5, the CPU 110 determines whether or not the approach of thefinger to the first touch panel 104B or the second touch panel 105B isdetected.

If, for example, the approach of the finger to the first touch panel104B is detected and a positive result is obtained in step SP5, the CPU110 transitions to step SP6.

In step SP6, the CPU 110 detects an approach operation and transitionsto next step SP7. In step SP7, the CPU 110 determines whether or not theapproach operation (for example, the above-described approach pinch-out,approach shuffle, or the like) corresponding to the operation input isdetected in step SP6 of the previous stage.

If the approach operation corresponding to the operation input isdetected in step SP6 of the previous stage and thus a positive result isobtained in step SP7, the CPU 110 transitions to step SP8.

In step SP8, the CPU 110 determines whether or not the current time iswithin the predetermined time T from the time when the touch operationcorresponding to the previous operation input is detected, based on theoutput of the timer.

If the current time is within the predetermined time T from the timewhen the touch operation corresponding to the previous operation inputis detected and thus a positive result is obtained in step SP8, the CPU110 transitions to step SP9.

In step SP9, the CPU 110 recognizes and receives the detected approachoperation as the series of operation inputs continued to the previoustouch operation and issues a command (for example, a command forreturning the display scaling factor of the page image to a value beforereduction) corresponding to the operation input. As a result, a process(for example, a process of returning the display scaling factor of thepage image to the value before reduction) corresponding to the commandis executed. Then, the CPU 110 resets the timer and returns to step SP1again.

If the current time exceeds the predetermined time T from the time whenthe touch operation corresponding to the previous operation input isdetected and thus a negative result is obtained in the above-describedstep SP8, the CPU 110 transitions to step SP10.

In step SP10, the CPU 110 recognizes and receives the detected approachoperation as an independent operation input and issues a command (forexample, a command for returning to a previous page) corresponding tothe operation input. As a result, a process (for example, a process ofreturning to the previous page) corresponding to the command isexecuted. Then, the CPU 110 resets the timer and then returns to stepSP1 again. If a negative result is obtained in the above-described stepSP7 (that is, if the approach operation detected in step SP6 is not theapproach operation corresponding to the operation input), the CPU 110returns to step SP1 again.

The CPU 110 of the mobile terminal 100 receives the touch operation andthe approach operation as the operation input by the operation inputreception procedure RT1.

1-5. Operation and Effect

In the above configuration, the CPU 110 of the mobile terminal 100monitors the output values of the capacitance type first touch panel104B and second touch panel 105B so as to recognize the touch operationand the approach operation thereof.

If the touch operation for the first touch panel 104B or the secondtouch panel 105B is detected, the CPU 110 recognizes and receives thetouch operation as the operation input and executes the processcorresponding to the operation input.

If the approach operation for the first touch panel 104B or the secondtouch panel 105B is detected within the predetermined time T after thetouch operation is detected, the CPU 110 recognizes and receives theapproach operation as the series operation input continued to theprevious touch operation.

In contrast, if the approach operation is detected when exceeding thepredetermined time T after the touch operation is detected, the CPU 110recognizes and receives the approach operation as the operation inputindependent of the previous touch operation.

Accordingly, in the mobile terminal 100, by recognizing and receivingthe approach operation as the operation input in addition to the touchoperation, it is possible to increase the types of the operation inputsand to perform various operation inputs, compared with the apparatus forperforming only the touch operation as the operation input.

In the mobile terminal 100, the approach operation is recognized andreceived as the operation input in addition to the touch operation. Inaddition, when the touch operation and the approach operation arecontinuously performed within the predetermined time T, these operationsare recognized as a series of operation inputs.

Therefore, in the mobile terminal 100, it is possible to recognize andreceive the touch operation and the approach operation as individualoperation inputs and to perform an operation input obtained by combiningsuch operations. As a result, it is possible to perform variousoperation inputs.

In the mobile terminal 100, the approach operation is recognized andreceived as the operation input in a state in which the general touchoperation (pinch-in, pinch-out, drag, tap, or the like) of the relatedart is added to such a touch operation without change.

Therefore, the mobile terminal 100 may be easily operated even by a userwho is familiar with the general touch operation of the related art. Asa result, various operation inputs and easy operation are compatible.

By the above configuration, since the mobile terminal 100 recognizes thetouch operation and the approach operation which are continuouslyperformed within the predetermined time T as a series of operationinputs, it is possible to increase the types of the operation inputs,compared with the case where only the touch operation is performed asthe operation input. Accordingly, the mobile terminal 100 may performvarious operation inputs compared with the related art.

2. Other Embodiment 2-1. Other Embodiment 1

In the above-described embodiment, within the predetermined time T, whenthe approach shuffle is performed after the contact pinch-in or contactpinch-out or when the approach pinch-out is performed after the contactpinch-out, these operations are received as the series of operationinputs.

The present invention is not limited thereto and a combination of theother various touch operations and approach operations may be receivedas the series of operation inputs.

For example, as shown in FIG. 12, the case where approach hold isperformed after contact drag will be described. Contact drag refers to atouch operation for touching one finger on a touch panel and moving(that is, sliding) the finger in a certain direction. Approach hold isan approach operation for approaching a touch panel with one finger andstopping the finger.

For example, in a state in which a part of a list of artist namesacquired from music data is displayed on the first touch screen 104, theuser performs the contact drag with respect to the first touch screen104.

When the contact drag is detected, the CPU 110 recognizes and receivesthe contact drag as an operation input for scrolling the list andscrolls the list according to the speed, distance and direction of thecontact drag.

Thereafter, it is assumed that the user separates the finger from thefirst touch screen 104 within the predetermined time T and performs theapproach hold.

When the approach hold is detected, the CPU 110 recognizes and receivesthe approach hold as an operation input for continuing to scroll thelist by the previous contact drag and continues to scroll the list.

At this time, the CPU 110 continues to scroll the list at the same speedand in the same direction as the scroll of the list by the previouscontact drag, during the approach hold.

Accordingly, for example, if the list is scrolled by several screens,the drag is repeatedly performed several times in the related art.However, in the mobile terminal 100, since only one contact drag andapproach hold is performed, it is possible to reduce user burden.

When the approach operation for separating and stopping the finger isperformed after the touch operation, a process (for example, scrolling)performed by the touch operation may be continuously performed.

2-2. Other Embodiment 2

In the above-described embodiment, when the approach operation isperformed within the predetermined time T after the touch operation,these are received as the series of operation inputs. The presentinvention is not limited thereto. When the touch operation is performedwithin the predetermined time T after the approach operation, these maybe received as the series of operation inputs.

For example, as shown in FIG. 13, the case where a tap is performed asthe touch operation after approach drag as the approach operation willbe described. Approach drag refers to an approach operation forapproaching a touch panel with one finger and moving the finger in acertain direction.

It is assumed that a software keyboard is displayed on the first touchscreen 104 or the second touch screen 105. In keys configuring thesoftware keyboard, keys for displaying a plurality of inputtablecharacters are included.

For example, in a key (this is also referred to as an M key) for mainlyinputting “M”, “M” is displayed in a center thereof, “?” is displayed inan upper end thereof, and “,” is displayed at a left end thereof.

At this time, it is assumed that, for example, the user moves theirfinger by a predetermined amount in an upper direction of the screen (aY direction of the drawing) on the M key, performs approach drag forreturning to an original state, and then taps the M key within thepredetermined time T.

When the approach drag and the tap are continuously detected, the CPU110 recognizes and receives the approach drag and the tap as the seriesof operation inputs for inputting “?” displayed on the upper end of theM key and issues a command for inputting the character “?”. As a result,the character “?” is input to the mobile terminal 100.

In contrast, it is assumed that the user moves the finger by apredetermined amount in a left direction (an X direction of the drawing)of the screen on the M key, performs approach drag for returning to anoriginal state, and taps the M key within the predetermined time T.

When the approach drag and the tap are continuously detected, the CPU110 recognizes and receives the approach drag and the tap as the seriesof operation inputs for inputting “,” displayed on the left end of the Mkey and issues a command for inputting the character “,”. As a result,the character “,” is input to the mobile terminal 100.

It is assumed that the user does not perform the approach drag on the Mkey and taps the M key.

In this case, when the tap is detected, the CPU 110 recognizes andreceives the tap as the operation input for inputting “M” displayed onthe center of the M key and issues a command for inputting the character“M”. As a result, the character “M” is input to the mobile terminal 100.

In the mobile terminal 100, by performing the approach drag in the upperdirection or the left direction just before tapping the M key, it ispossible to selectively input the character other than “M” allocated tothe M key.

Accordingly, in the mobile terminal 100, it is possible to input aplurality of characters using one key by combination of the direction ofthe approach drag and the tap. Therefore, it is possible to reduce thenumber of keys displayed on the screen and, as a result, to efficientlyuse the screen. In addition, it is possible to realize an intuitivelysimpler operation, compared with the case where a shift key or the likeis touched so as to switch a character inputtable by each key.

The present invention is not limited to such a character input, and thecontinuous operation of the approach drag and the tap may be the othervarious operations and may be applied to, for example, a web browseroperation.

In practice, for example, an upper half of the web browser screen isdisplayed on the first touch screen 104 and a lower half of the webbrowser screen is displayed on the second touch screen 105.

At this time, as shown in FIG. 14, it is assumed that the user moves thefinger by a predetermined amount in a right direction (an X direction ofthe drawing) of the screen on a link displayed on the lower half of theweb browser screen, performs approach drag for returning to an originalstate, and taps the link within the predetermined time T.

When the approach drag and the tap are continuously detected, the CPU110 recognizes and receives the approach drag and the tap as a series ofoperation inputs for opening a page of a link destination by the tap andissues a command for opening the page of the link destination by thetap. As a result, on the first touch screen 104 and the second touchscreen 105, the page image of the page of the link destination isdisplayed by the tap.

In contrast, as shown in FIG. 15, it is assumed that the user movestheir finger by a predetermined amount in the upper direction (the Ydirection of the drawing) of the screen on the link displayed in thelower half of the web browser screen, performs approach drag forreturning to an original state, and taps the link within thepredetermined time T.

When the approach drag and the tap are continuously detected, the CPU110 recognizes and receives the approach drag and the tap as a series ofoperation inputs for opening a page of a link destination on an upperscreen and issues a command for opening the page of the link destinationon the upper screen. As a result, the display content of the secondtouch screen 105 is not changed and the page image of the page of thelink destination is displayed only on the first touch screen 104.

In the mobile terminal 100, by performing the approach drag in the upperdirection or the right direction just before tapping the link, it ispossible to open the page of the link destination using a certainopening method (whether the page is opened on the upper screen or isopened by the tap).

In the mobile terminal 100, a combination of the touch operation and theapproach operation received as the series of operation inputs isregistered and, when the touch operation and the approach operation areperformed by the registered combination, these operations are receivedas the series of operation inputs.

Such a combination and the operation input corresponding to thecombination may be registered by the user. Thus, the user may customizethe operation of the mobile terminal 100 so as to facilitate the use ofthe mobile terminal.

2-3. Other Embodiment 3

In the above-described embodiment, when the approach operation isperformed within the predetermined time T after the touch operation forthe first touch screen 104 or the second touch screen 105, theseoperations are received as the series of operation inputs.

The present invention is not limited thereto and, for example, when theapproach operation for a touch screen subjected to the touch operationis performed within the predetermined time T after the touch operation,these operations may be received as the series of operation inputs.

In addition, for example, when the approach operation is performedwithin a predetermined range centered on a touch position subjected tothe touch operation after the touch operation, these operations may berecognized and received as the series of operation inputs.

2-4. Other Embodiment 4

In the above-described embodiment, when a touch operation and anapproach operation are continuously performed within the predeterminedtime T without depending on which operation is performed, theseoperations are received as the series of operation inputs.

The predetermined time T is only exemplary and the point is to receivethe operations as the series of operation inputs when a touch operationand an approach operation are continuously performed within the setpredetermined time. The present invention is not limited thereto and thepredetermined time T may be set with respect to each operation. Forexample, the predetermined time T may be set to 5 seconds after contactpinch-out and 1 second after contact drag.

The present invention is not limited thereto. For example, thepredetermined time T may not be set and, when specific touch operationand approach operation are continuously performed, these operations maybe recognized and received as the series of operation inputs.

2-5. Other Embodiment 5

In the above-described embodiment, the present invention is applied tothe mobile terminal 100 having the capacitance type first touch panel104B and second touch panel 105B.

The present invention is not limited thereto. The present invention isapplicable to an apparatus including the other various operation inputdevices, such as an optical sensing touch screen including an opticalsensor in a liquid crystal panel, if it is an operation input devicecapable of detecting the contact and approach of a finger (indicator).

The present invention is applicable to an apparatus including the othervarious display devices, such as an organic Electro Luminescence (EL)panel, instead of the first liquid crystal panel 104A and the secondliquid crystal panel 105A.

2-6. Other Embodiment 6

In the above-described embodiment, the present invention is applied tothe mobile terminal 100 having two screens, that is, the first touchscreen 104 and the second touch screen 105.

The present invention is not limited thereto and the present inventionis applicable to, for example, an apparatus having one screen, that is,one touch screen.

2-7. Other Embodiment 7

In the above-described embodiment, the first touch panel 104B and thesecond touch panel 105B functioning as the contact detection unit 2 andthe approach detection unit 3 are provided in the mobile terminal 100 asthe information processing device 1. The CPU 110 is provided as thecontrol unit 4 of the information processing device 1.

The present invention is not limited thereto. If the same function isperformed, each unit of the above-described mobile terminal 100 may beconfigured by a variety of other hardware or software.

The present invention is not limited to the mobile terminal 100 and isapplicable to various apparatuses such as a digital still camera, astationary personal computer, a games machine, a mobile audio player ora mobile telephone.

2-8. Other Embodiment 8

In the above-described embodiment, a program for executing a variety ofprocesses is written in the non-volatile memory 111 of the mobileterminal 100.

The present invention is not limited thereto. For example, a slot of astorage medium such as a memory card may be provided in the mobileterminal 100 and the CPU 110 may read and execute a program from thestorage medium inserted into the slot. In addition, the CPU 110 mayinstall the program read from the storage medium in the non-volatilememory 111. In addition, the CPU 110 may download the program from anapparatus on a network through a network I/F 114 and install the programin the non-volatile memory 111.

2-9. Other Embodiment 9

In addition, the present invention is not limited to the above-describedembodiment and the other embodiments. That is, the present invention isapplicable to an arbitrary combination of a part or all of theabove-described embodiment and the other embodiments or a part extractedfrom the above-described embodiment and the other embodiments.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An information processing device comprising: acontact detection unit configured to detect a contact operation for anoperation surface; an approach detection unit configured to detect anapproach operation for the operation surface; and a control unitconfigured to recognize the contact operation and the approach operationas a series of operation inputs when the contact operation and theapproach operation are continuously detected.